Skip to content

[HUDI-6516] Correct the use of hoodie.bootstrap.mode.selector #9164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
codope merged 2 commits into from
Jul 23, 2023
Merged

[HUDI-6516] Correct the use of hoodie.bootstrap.mode.selector #9164

codope merged 2 commits into from
Jul 23, 2023

Conversation

@Zouxxyy
Copy link
Contributor

@Zouxxyy Zouxxyy commented Jul 10, 2023 •
edited
Loading

Change Logs

The following is the current configuration of bootstrap's mode selection:

  • hoodie.bootstrap.mode.selector
  • hoodie.bootstrap.mode.selector.regex
  • hoodie.bootstrap.mode.selector.regex.mode

Fix #6673 (comment), The correct use of hoodie.bootstrap.mode.selector is:

  • set hoodie.bootstrap.mode.selector as BootstrapRegexModeSelector, specify partitions by hoodie.bootstrap.mode.selector.regex, then set hoodie.bootstrap.mode.selector.regex.mode for the specify partitions

  • set hoodie.bootstrap.mode.selector as MetadataOnlyBootstrapModeSelector or FullRecordBootstrapModeSelector

Impact

Correct the use of hoodie.bootstrap.mode.selector

Risk level (write none, low medium or high below)

low

Documentation Update

none

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Jul 11, 2023

CI failure is caused by other, Can you help with a review? @codope

@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Jul 22, 2023 •
edited
Loading

@danny0405 @codope Can someone help with a review? It's a bug fix, before it the usage and test cases of hoodie.bootstrap.mode.selector are wrong. And I've already resolved one conflict...

codope
codope reviewed Jul 23, 2023
View reviewed changes
Copy link
Member

@codope codope left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zouxxyy Can you tell me what scneario is fixed by this PR?
As per my understanding, hoodie.bootstrap.mode.selector can take one of three classnames:

  1. BootstrapRegexModeSelector
  2. MetadataOnlyBootstrapModeSelector
  3. FullRecordBootstrapModeSelector
    If that config is set to BootstrapRegexModeSelector then hoodie.bootstrap.mode.selector.regex is used (default value .* means all partitions).
    And then there is a config hoodie.bootstrap.mode.selector.regex.mode which can be either FULL_RECORD or METADATA_ONLY. This config cannot be set to METADATA_ONLY if we are using FullRecordBootstrapModeSelector. The current logic seems to cover different scenarios. Are we missing a corner case?

@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Jul 23, 2023

@codope

And then there is a config hoodie.bootstrap.mode.selector.regex.mode which can be either FULL_RECORD or METADATA_ONLY. This config cannot be set to METADATA_ONLY if we are using FullRecordBootstrapModeSelector.

see BootstrapRegexModeSelector
image

hoodie.bootstrap.mode.selector.regex.mode is only used with BootstrapRegexModeSelector, It has nothing to do with MetadataOnlyBootstrapModeSelector or FullRecordBootstrapModeSelector

@apache apache deleted a comment from hudi-bot Jul 23, 2023
@hudi-bot
Copy link
Collaborator

hudi-bot commented Jul 23, 2023

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@codope codope merged commit c14ac0e into apache:master Jul 23, 2023
@hudi-bot hudi-bot mentioned this pull request Dec 9, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@codope codope codope approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Zouxxyy @hudi-bot @codope